[docs] Use other checkpoints with inpaint#5590
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
patrickvonplaten
left a comment
patrickvonplaten
left a comment
There was a problem hiding this comment.
It would be good if we can write a whole section about it here and also show how it differs, what options can be enabled by the user cc @yiyixuxu maybe you can help here a bit as well?
|
I expanded more on the section to show how quality differs for more basic vs complex tasks. Feel free to add anything I may have missed @yiyixuxu 🙏 |
yiyixuxu
left a comment
yiyixuxu
left a comment
There was a problem hiding this comment.
great!
I think we should talk about the other side of trade-off here. Basically the official in-paint pipeline has better generation qualities but it is less "respectful" to the mask. It means that it's much more willing to change the content around the mask edge in order to generate a better looking image. If you want your unmasked area fixed, you can't use the official inpaint pipeline
on the other hand, if we use our regular test-to-image unet to generate inpaint. at each denoising step, we will paste over the original image latent to the unmasked area, so it is more respectful to the mask, so much so that you often can see the mask edge. And the overall image is less natural. But if you need to have your unmaksed area fixed, you can only achieve that with non-inpaint specific checkpoint. (we have code in the doc that teaches how to do that I think). and this PR may give you a little bit more info on this #3880
|
Thanks for the excellent explanation @yiyixuxu 🙇♂️ , updated the docs to discuss the trade-off and merged the "preserve unmasked areas" code with this section |
yiyixuxu
left a comment
yiyixuxu
left a comment
There was a problem hiding this comment.
looks great:) thank you
* tip about inpaint checkpoints * expand section * feedback
4 participants
Adds a tip clarifying you can use non inpaint-specific checkpoints in the inpainting pipeline.